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Foreword 

The publication of Effective Self-evaluation Reporting in Scotland’s Colleges follows 
the publication in August 2007 of the final report of the Joint Quality Review Group 
(JQRG) of the Scottish Funding Council to Council members.^ The JQRG report: 

‘sets out the Joint Quality Review Group’s final proposals for quality 
enhancement and assurance processes in the college and university sectors.’ 

The overall model for the quality system proposed in the JQRG report is based on 
the three key principles of: 

■ high quality learning across all provision; 

■ student engagement; and 

■ a culture of quality and continuous improvement. 

It is intended that these be delivered through three key areas of activity which build 
on the effective practices which already exist within both the university and college 
sectors. These are: 

■ college and university-led evaluation; 

■ external review; and 

■ Scotland-wide enhancement activities. 

The JQRG report places considerable emphasis on the role of institution-led quality 
processes. 

‘Institutional culture is the most important element in our system for enhancing 
and assuring quality. A pervasive and embedded commitment to reflecting on 
and improving what institutions do, how they support learners, how they 
engage with students, and the standards they maintain, is essential to 
achieving our goals for quality. External review cannot and should not take 
the place of institutional evaluation, reflection and action. It is essential that 
every aspect of our approach continues to emphasise the responsibility, 
ownership and accountability for quality that falls to institutions.’ 

The findings and recommendations of Effective Self-evaluation Reporting in 
Scotland’s Colleges are central to discussions in the college sector about how to 
carry out self-evaluation as effectively as possible, and further develop effective 
institutional quality cultures, one of the key principles of the JQRG report. The 
findings and recommendations should therefore be considered in tandem with the 
JQRG report’s focus on the role of internal quality enhancement and assurance 
systems in meeting the needs of learners. Its findings and recommendations will 
contribute to the consultation process on the JQRG report, due to be completed by 
Spring 2008. 


* Final report from the Joint Quality Review Group to Council, 2007, 

www.sfc.ac.uk/about/new_about_council_papers/about_papers_17aug07/SFC_07_l 1 3_ANNEX.pdf - 2007-08- 
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1. Introduction 

Self-evaluation is one of the cornerstones of quality in Scotland's colleges. It is a 
key process used by colleges to improve continuously the services they provide for 
learners and other stakeholders. Its purpose is to help college staff evaluate 
evidence in order to identify: 

■ what they are doing well; 

■ any weaknesses they need to address: and 

■ what they have to do to make a good experience for learners even better. 

It aims to create a sense of ownership in staff for actions to improve the quality of the 
services they provide to learners and therefore a greater commitment to ensuring 
that quality improvement actually does take place. 

Self-evaluation is broadly defined to include any process which helps college staff, 
individually or collectively, to achieve the above aim with input from learners and 
other stakeholders. It includes activities such as: 

■ obtaining the views of learners and other stakeholders; 

■ collating and analysing performance indicators (Pis) for learner outcomes 
such as retention, achievement, attainment and progression; 

■ identifying trends and using benchmark information; 

■ analysing key messages from the external and internal operating 
environment; 

■ considering the findings of external and internal moderation and internal 
audits; 

■ observing learning and teaching; 

■ engaging in professional reflection and discussions both formally and 
informally. 

The outcomes of self-evaluation include: 

■ identification of weaknesses that need to be addressed; 

■ identification of good practice for dissemination; 

■ information on the quality of provision for college staff, senior managers 
and boards of management; 

■ intelligence that helps college staff and managers to plan for the future. 
Self-evaluation is followed by: 

■ drawing up plans for improvement; 

■ monitoring the achievement of targets and the impact of actions; 

■ dissemination and promotion of good practice; and 

■ giving assurance on the quality of provision, including affirmation and 
celebration of practice where appropriate. 

Although college staff carry out and integrate the above activities throughout the 
year, most are also involved in a more formal self-evaluation reporting process. This 
process usually takes place annually. Its characteristics are that it is carried out in 
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teams and it evaluates evidence from the activities listed above against a quality 
framework. It culminates in a written, graded self-evaluation report which includes a 
quality improvement action plan. This is the recorded summary of all self-evaluative 
activities. The different ways colleges carry out this formal process are detailed in 
section 4. For the purposes of this report we have defined this formal process as 
self-evaluation reporting, and it is the main focus of the report. 

The main purpose of this process is still, of course, to facilitate quality improvement 
in colleges. This includes mechanisms for colleges to enhance quality by identifying 
and disseminating good practice uncovered through the self-evaluation process. 
However, the self-evaluation reporting process has an important further purpose. 
Through the production of self-evaluation reports, it can provide assurance to senior 
managers and, in some cases, boards of management that services to learners are 
being properly monitored, evaluated and improved by college staff. Self-evaluation 
reports are almost always only disseminated within colleges at present. They are 
made available to HMIE prior to external review but are not routinely disseminated 
either to external stakeholders or to learners. However, ratification at senior 
management levels within colleges, together with evaluation by HMIE of the 
effectiveness of a college’s self-evaluation processes provides a degree of 
assurance to stakeholders that college staff take improvement seriously. The 
interconnecting roles of quality improvement and quality assurance make 
self-evaluation reporting a vital component of the interface between organisational 
capacity building and accountability. 

In order to make a distinction between self-evaluation reporting and other ongoing 
self-evaluation processes which take place outwith the formal one, this report refers 
to these wider processes as self-evaluation activities. Amongst these activities, 
instances where staff meet and discuss what they have been doing are defined in 
the report as professional dialogue. This includes, for example, team meetings, 
staff review and development interviews, internal moderation discussions or the 
conversations which follow peer observation. Finally, the report defines as 
professional reflection the occasions where individual college staff consider the 
impact of their own performance and use this insight to improve their practice. This 
might take place, for example, when a member of staff has attended a training 
session, observed colleagues in action or read a professional journal. There is of 
course, a clear relationship between professional reflection, professional dialogue, 
wider self-evaluation activities and the formal self-evaluation reporting process, as 
the diagram below demonstrates. 
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Colleges carry out self-evaluation reporting in teams. In this report these teams are 
defined in three groups. Teaching teams are those with direct involvement in 
curriculum delivery, but they may include non-teaching staff. Support teams are 
those which provide all other services within the college. College management 
teams are those involved in college-wide self-evaluation. These teams may include 
senior and middle managers. The report uses the term college teams when all 
three groups are being referred to. 

The publication in 1998 by HMIE of Quality Matters: Quality Improvement through 
Self-evaluation,^ marked the formal beginning of self-evaluation against a quality 
framework in all of Scotland’s colleges, although a draft version had been issued a 
year before. This publication was expected to lead to colleges carrying out their own 
internal review through self-evaluation with the same degree of rigour as HMIE. It 
promoted internal ownership of quality improvement, with HMIE providing an 
independent, external view of college provision against which colleges could validate 
their self-evaluation. From 1998, most colleges embarked on self-evaluation by 
devising procedures that built upon their existing quality processes. Today, 
self-evaluation is a well-embedded process that involves most college staff in some 
way. It is carried out as a matter of established practice in curriculum areas, and 
increasingly, in support areas. Colleges treat it seriously and strive to support and 
improve it. 


^ Quality Matters: Quality Improvement Through Self-evaluation, (Edinburgh) The Scottish Office, 1998 
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The HMIE publication, Improving Scottish Education (2006)^ commented: 

In recent years most colleges have demonstrated increased maturity in 
developing and implementing systems and procedures to help deliver 
effective learning experiences for a wide and expanding range of learners and 
to meet the needs of society and the economy. Where programme teams 
have implemented rigorous and well-informed self-evaluation processes, this 
has often led to improvements in the learner experience. However, 
weaknesses are still prevalent in the approaches of some programme teams 
and colleges to quality assurance and improvement activities.' 

Currently, HMIE expects self-evaluation reporting to be in place in Scotland's 
colleges but external review teams do not explicitly seek to test the validity of 
self-evaluation reports, as happens in some other education review models in 
Scotland and the rest of the UK."^ However, HMIE does explicitly compare 
self-evaluation grades with review grades and provides colleges with written 
summaries of strengths and weaknesses for all elements reviewed.^ 

This report analyses what different stakeholders within the college sector expect of 
self-evaluation reporting almost ten years after its inception, and identifies what they 
believe needs to be in place to make it more effective. It also draws on relevant 
published reports. It evaluates whether self-evaluation reporting as it is currently 
carried out in colleges fulfils all these collectively-held expectations and whether it 
really does help college staff effect substantial improvements in services to learners. 
It also cites examples of practice that have a positive impact on self-evaluation 
reporting. 


^ Improving Scottish Education, HMIE 2006 
'* Examples of these models can be found in appendix 1 

^ Analysis of concurrence between HMIE and self-evaluation grades is detailed in appendix 4 
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2. Methodology 

The report draws on evidence from the analysis of HMIE reports on 29 college 
reviews that have taken place since January 2005, especially those sections on 
quality assurance, quality improvement, learning and teaching and learner progress 
and outcomes. Desk research on reports and literature from the college and higher 
education institution (HEI) sectors in Scotland and the rest of the UK, and 
information about other quality systems, formed essential background evidence for 
the report. 

In addition, inspectors visited six colleges and held structured interviews with senior 
managers, department heads and team leaders, teaching and support staff and 
learners. Interviews also took place with senior managers in four other colleges. 
Telephone and face-to-face interviews took place with key representatives from the 
college and HEI sectors. Self-evaluation reports and other quality documentation 
collected for college reviews and provided by other colleges were analysed. HMIE 
held focus group sessions on early messages from this report with representatives 
from seven out of the ten colleges visited during the fieldwork for the report. Similar 
discussions took place with members of the Scottish Further Education Unit (SFEU) 
Quality Community of Practice (formerly the Quality Improvement Forum). A full list 
of those colleges and organisations consulted is shown in Appendix 2. 
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3. Summary of findings 

Most learners enjoy a positive learning experience in Scotland’s colleges and most 
achieve their learning aims. Analyses of review reports by HMIE demonstrate that 
overall the quality of the learner experience is improving, though there are still areas 
which colleges need to address. 

Self-evaluation activities and self-evaluation reporting are well embedded into quality 
processes in Scotland’s colleges. Almost all college staff are familiar with the 
self-evaluation reporting process and most have been involved with it, especially in 
teaching teams. 

Self-evaluation reporting has a positive impact overall on improvement in services to 
learners. It has encouraged a culture of reflection and questioning amongst college 
staff, and underpins a focus on continuous improvement. 

Positive attitudes overall to self-evaluation reporting in Scotland’s colleges provide a 
sound basis on which to develop it further and give it even more prominence within 
quality processes. Almost all colleges are engaged in improving the effectiveness of 
self-evaluation reporting through involvement of external development agencies or 
through internal support, training and guidelines. They are also committed to 
developing skills in professional reflection and professional dialogue. 

Self-evaluation reporting is most effective when it builds on and reports the findings 
and impact of professional dialogue and other self-evaluation activities. Colleges do 
not always make this linkage well. Self-evaluation reporting is least effective when it 
is carried out in isolation from these other processes. 

When college staff are actively involved in continuously improving provision through 
ongoing professional dialogue which underpins and runs parallel to self-evaluation 
reporting, many express the view that professional dialogue has more direct impact 
on improvement in learning and teaching than formal self-evaluation reporting. 
Professional dialogue provides college staff with more immediate and private 
mechanisms for addressing concerns. However, because evidence of these 
discussions is not generally well documented in self-evaluation reports, opportunities 
to provide assurance that they are taking place are often missed by colleges. 

Most colleges support self-evaluation reporting by providing their teams with robust 
performance data and by implementing effective monitoring systems. Most provide 
helpful guidelines and support for self-evaluation reporting and also synchronise 
self-evaluation reporting cycles and operational planning cycles well. 

Over-prescriptive approaches to the use of quality frameworks constrain the capacity 
of college staff to contextualise self-evaluation reporting, or to generate holistic, 
innovative approaches to quality improvement. 

The quality of self-evaluation reports has generally improved over the last four years, 
but still varies widely. A few provide very full, well-evidenced evaluations of services 
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and their impact on iearners and inciude ciear action pians with SMART® targets. 
Most, however, are descriptive rather than evaiuative and do not anaiyse evidence 
rigorousiy enough. They are insufficientiy focused on iearning and teaching, 
retention and attainment, or on team performance. They are aiso insufficientiy ciear 
or detaiied. This means that currentiy, seif-evaiuation reporting in Scotiand's 
coiieges is not aiways carried out as weii as it couid be. it aiso means that many 
coiieges are unabie to provide assurance through their seif-evaiuation reports that aii 
teams are effective at continuousiy improving their services to iearners. 

Action pianning foiiowing seif-evaiuation reporting is mostiy insufficientiy detaiied and 
not comprehensive. Most action pian targets are insufficientiy specific or 
measurabie. This is particuiariy evident around iearning and teaching, and retention 
and attainment, and usuaiiy resuits from insufficient identification or evaiuation of 
these issues in the seif-evaiuation reports. 


® Specific, Measurable, Achievable, Realistic, Timebound 
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4. How colleges currently carry out self-evaluation reporting 

All of Scotland’s 43 colleges carry out self-evaluation reporting, in virtually all 
teaching teams, and, increasingly, in support and college management teams. Each 
college has devised its own way of carrying out self-evaluation reporting to suit its 
internal planning cycles and organisational structures. 

All colleges carry out self-evaluation reporting based on external quality frameworks. 
Teaching teams currently use the nine quality elements in Area A of the SFC/HMIE 
quality framework of 2004, curriculum resources, processes and outcomes. These 
are commonly known as the ‘A’ elements. Self-evaluation reporting by college 
management teams focuses on Area B of the SFC/HMIE quality framework, 
leadership and quality management. This area has seven quality elements, known 
as the ‘B’ elements.^ 

Support teams also use or adapt the SFC/HMIE framework and, in many cases, also 
use other external self-evaluation reporting toolkits. Those most commonly used 
include: 

■ Resources and Services Supporting Learning for learning resource teams;® 

■ Bootstrap for ICT support teams;® and 

■ Bursaries and College Managed Student Support Funds for college bursary 
teams. 

SFEU was commissioned by the Quality Community of Practice to consult with 
college support staff and drawn up a set of generic indicators for support areas. 
Some colleges have adopted this set of indicators for use alongside the SFC/HMIE 
framework. 

Many colleges also use other quality frameworks to evaluate different aspects of 
their services. Chief among these is Investors in People^^, which focuses on people 
management and people development and Charter Mark^^, the UK Government's 
national standard for excellence in customer service. A small number of colleges 
also carry out self-evaluation using the European Foundation for Quality 
Management (EFQM) Excellence Model. 

All colleges carry out self-evaluation reporting cyclically. Because of the emphasis in 
HMIE external review on learning and teaching process, (A5 in the framework) and 
learner progress and outcomes (A7), most teaching teams carry out self-evaluation 
reporting using at least A5 and A7 in a yearly cycle. The other ‘A’ elements may be 


^The framework elements are listed in Appendix 3 

^Resources and Services Supporting Learning: a Service Development Quality Toolkit, SFC/SLIC 2003, 
www.slainte.org.uk/slic/index.htm; 

^Bootstrap - The ICT Services Toolkit, Jisc infonet, 2004, 
www.jiscinfonet.ac.uk/Resources/evalkit/toolkit-database/ev006; 

''’Bursaries and College Managed Student Support Funds Self-evaluation Toolkit, SFEU, 
www.sfeu.ac.uk/search/Bursaries+and-HCollege-HManaged-tStudent-tSupport-i-FundsH-Self-evaluation-t-Toolkit 
” Developing a Framework for the Self-evaluation of Support Services, SFEU, 2006 
www.investorsinpeople.co.uk/About/Pages/roleofiipuk.aspx 
www.cabinetoffice.gov.uk/chartermark/ 
see appendix 1 
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reviewed either every year, every two years or only where they impact significantly 
on the learning and teaching process (A5) or learner progress and outcomes (A7). 
Support team self-evaluation reporting also varies in frequency, with the most 
common frequency being once per year. Colleges also vary in how frequently they 
carry out college-wide self-evaluation reporting, but almost always carry it out prior to 
external review. Within a four-year cycle, almost all college teams that carry out 
self-evaluation reporting will have covered all aspects of their provision at least once. 

Although HMIE does not specify exactly how quality frameworks should be used, 
there is an expectation that a college will have used the SFC/HMIE framework to 
some extent in its self-evaluation reporting prior to external review. In almost all 
cases, college teams award each element in their reports an evaluation grade based 
on the same four-point scale used by HMIE for external reviews. These are: 

Very good - major strengths 

Good - strengths outweigh weaknesses 

Fair - some important weaknesses 

Unsatisfactory - major weaknesses 

College organisational structures impact on how self-evaluation reporting is carried 
out and by whom. Most colleges apply a degree of aggregation in departments 
(sometimes known as faculties or schools) which have a number of small teams. 
Teaching team self-evaluation reporting is most commonly led by the programme 
team leader, but in some cases quality framework elements or even preparation of 
the whole self-evaluation report is carried out by department heads. This is usually a 
reflection of the size of departments and the number of teams. 

Most support teams carry out self-evaluation reporting in their operational 
departments and there is less aggregation into larger departments. Where 
aggregation does occur it is most commonly where team reports are subsumed into 
the college self-evaluation reports (the ‘B’ elements). This happens most frequently 
to self-evaluation reports on learning resources, guidance, staffing or quality. 

College teams carry out self-evaluation reporting in different ways. In the majority of 
cases, self-evaluation reporting takes place with members of the team present. This 
might be at a specific set-aside time in which team members jointly construct the 
report, or it may be carried out incrementally during regular scheduled team 
meetings. In more than a few colleges the team leader will draft a report outwith a 
team meeting and circulate it to members for comment, or place it on a shared 
computer drive so that staff can contribute to it. Team leaders sometimes do this to 
capture the views of part-time or remotely situated staff members, even where a 
whole team meeting is planned. 

Most colleges attempt to schedule self-evaluation reporting to make best use of 
performance indicators or to ensure its findings can inform operational planning. 

Thus A7, learner progress and outcomes, is often evaluated in the autumn when the 
performance indicators for the previous year have been audited, and A5, learning 
and teaching process, is commonly evaluated in the spring when staff have had time 
to get to know current learners. 
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Almost all colleges operate a monitoring system for self-evaluation reporting and the 
progressing of quality improvement action plans. In some colleges, this is in the 
form of a written report from senior or quality managers on how well self-evaluation 
reporting has been carried out. In other colleges, it may take the form of face-to-face 
meetings between college teams and senior managers who monitor how well action 
plans are being achieved and discuss wider quality or planning issues. Many 
colleges have rigorous internal review processes which mirror the external ones 
operated by HMIE. 

Colleges have devised a diverse range of proformas for self-evaluation reporting. In 
some cases the quality framework prompts are reconfigured in the form of questions. 
In others, teams are asked to identify not only strengths and weaknesses but areas 
of good practice. Reports vary in style, from narrative form to bulleted headings, or a 
combination of both. 
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5. Shared understanding of self-evaluation reporting 

Summary of discussions with the coiiege sector 

There is a consensus that self-evaluation reporting has developed a great deal since 
its inception and is now well embedded into college quality processes. All colleges 
and sector representatives share the view that together with the wider self-evaluation 
activities that include professional reflection and dialogue, self-evaluation reporting 
has a major role to play in quality improvement in colleges, and should continue to 
be supported and developed. They understand the importance of the place of 
self-evaluation reporting in quality systems. 

Colleges and sector representatives are fully committed to the aims of 
self-evaluation activities and reporting, but recognise that now is a useful time to 
review how well their processes fulfil these aims. While the sector is clearly focused 
on continuous improvement in many different ways, there are variations in how well 
self-evaluation reporting is carried out. Some colleges are concerned that a few staff 
take part reluctantly or as a formal response to requirements, because they 
understand that self-evaluation reporting works best if it is valued by staff who are 
committed to it.. 

Scotland's colleges are highly complex environments and each one is different. 

They vary a great deal in size, and have a wide range of learners^® and 
programmes. Change is very rapid in colleges. They need to be able to 
continuously respond to this complex and changing environment in order to keep on 
attracting learners and fulfil their learning and support needs. Colleges recognise 
that self-evaluation reporting makes good use of environmental intelligence to 
improve provision and usefully support college strategic planning. Most importantly, 
it provides an agenda for change in the form of an action plan which has its origins in 
shared understanding of reality and, if carried through properly, leads colleges 
directly to improved services for learners. 

Well-led self-evaluation activities impact on staff in other useful ways, by 
underpinning the development of positive quality cultures. This is important because 
it maintains the focus of all staff on improvement and enhancement and helps them 
fulfil wider college aims. 

Discussions with college staff and sector representatives helped to define the 
characteristics of effective self-evaluation reporting set out in the next section. They 
cover how self-evaluation helps staff improve services to learners and what has to 
be in place to help staff make it work well. 


Improving Scottish Education, HMIE 2006; and 
Learning to Improve: Quality Approaches for Lifelong Learning, Scottish Executive, 2005 
** Unlocking Opportunity, The Difference Scotland’s Colleges Make to Learners, the Economy and Wider 
Society, Scottish Executive, 2006 
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6. Characteristics of effective seif-evaiuation reporting 

Characteristics of effective seif-evaiuation reporting are summarised beiow in the 
green boxes under six main themes. Foiiowing each theme is a summary of 
strengths and areas for deveiopment in Scotiand’s coiieges. Accounts of practice 
which has had a positive impact on seif-evaiuation in coiieges appear in the paie 
orange boxes after a number of these sections. 


6.1 Theme 1 : Team working 

Self-evaluation reporting is effective when it provides a mechanism for college 
staff to collectively identify, share and agree ways of improving and enhancing 
the services they provide. 


The collective aspect of self-evaluation reporting is particularly important. When effective 
self-evaluation activities and reporting take place amongst a group of people with a common 
purpose, they generate positive team interaction, creative thinking and collaborative, efficient 
approaches to improving and enhancing services for learners. These features encourage 
the whole team to take responsibility for outcomes and to feel positive about their work. In 
order to be effective self-evaluation reporting has to be well structured, wide ranging and 
include the views, knowledge and experiences of all members of the team. Staff are more 
likely to be open with each other and not feel threatened by self-evaluation reporting in 
colleges which have supportive working relationships and a non-blaming culture. The skills 
of leaders are critically important in facilitating self-evaluation reporting discussions, 
prioritising outcomes and drawing up action plans to which everyone has agreed.^^ 


Strengths 

Most coiieges try to invoive as many staff as possibie in seif-evaiuation reporting, 
either by inciuding them in the meetings convened to construct the report or by 
giving them the opportunity to comment on reports. Aimost aii coiiege managers and 
fuii-time curricuium staff, and increasing numbers of support staff, are invoived in 
these processes. 

The majority of team ieaders have deveioped usefui skiiis in constructing, ieading 
and managing seif-evaiuation reporting team meetings and carry out this part of the 
process weii. A few are exceptionaiiy weii organised, sensitive to underiying staff 
concerns, good at defusing contentious issues and good at extracting key messages 
from wide-ranging group discussions. 

Seif-evaiuation reporting team meetings are generaiiy productive when there is 
enough time aiiocated to them and when team reiationships are aiready positive, 
in most cases the production of the seif-evaiuation report heips to articuiate team 
concerns effectiveiy, inciuding coiiege-wide issues. 


PDA Advanced Diploma in Quality Improvement, SQA, 2007 
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When department reports aggregate themes from individual team reports or are the 
result of self-evaluation reporting on wider departmental themes, more than a few 
highlight major issues effectively and generate action plans that are common and 
applicable to all teams within the department. There are a few examples of 
well-constructed links between curriculum team self-evaluation reports and 
college-wide self-evaluation reports. 

In most cases quality improvement action plans are the result of agreement by team 
members, who therefore have a shared understanding of how to take the actions 
forward. 

Areas for development 

In most colleges, part-time staff and staff who work in outreach locations are less 
involved in self-evaluation activities and reporting than other staff. A few feel 
marginalised from the outcomes of these processes as a result. 

The majority of aggregated department reports are too general to provide action 
plans which are meaningful to their constituent teams. Cross-college self-evaluation 
reporting is not always carried out systematically and outcomes are rarely shared 
with all college staff, except occasionally prior to external review. 

Although most college staff find discussions in self-evaluation reporting team 
meetings stimulating and interesting, producing the report and action plan is less 
successful as a team activity because it is difficult for a group of people to 
systematically scan and collate all the relevant evidence. More than a few 
self-evaluation reports and quality improvement action plans are poorly structured 
and not fully comprehensive. 

It is rare for college management team or teaching team self-evaluation reports to be 
critical of the team’s own performance. This is particularly evident around the 
evaluation of learning and teaching. It is more common for support team 
self-evaluation reports to identify team actions which are based on critical comments 
from internal and external clients. 

During self-evaluation reporting team meetings, most team leaders find it difficult to 
address areas of underperformance and very difficult to record them in 
self-evaluation reports. A few of these team leaders even feel that their loyalty to the 
team would be questioned by team members if they reported team areas for 
improvement too openly. 
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SET-ASIDE TIME 

Several colleges carry out self-evaluation reporting on days when all staff can work 
together without learners present. This allows staff to dedicate time to the process, 
without interruption. It also means that staff can work in cross-curricular and 
cross-college teams. In some cases colleges ensure that part-time staff can also 
attend these sessions. Quality managers are able to offer on-the-spot support and 
senior managers can participate in action planning with staff. 


6.2 Theme 2: Commitment of college teams 

Self-evaluation reporting is effective when college staff value it as a means of 
continuously improving services for learners. 


College commitment to carrying out seif-evaiuation reporting properiy is criticai in order for it 
to have vaiue, otherwise it can become an exercise in compiiance. Staff who vaiue the 
process are more iikeiy to carry it out thoroughiy and honestiy and are more iikeiy to foiiow 
through on resuiting quaiity improvement action pians. The attitudes and skiiis of team 
ieaders in particuiar, have a significant infiuence on the attitudes of team members, and 
therefore the outcomes of seif-evaiuation reporting. Teams vaiue the process more when 
they have scope to contextuaiise seif-evaiuation reporting and adapt quaiity frameworks so 
that they are reievant to their iearners’ specific needs. This provides teams with actions 
which have reai meaning to their members.^^ The operationai ievei of the teams carrying 
out seif-evaiuation activities infiuences the ievei of detaii of the reports and how fuiiy staff are 
committed to impiementing action pians. Staff vaiue seif-evaiuation reporting more when 
senior managers vaiue it, are assured by it and foiiow through on coiiege-wide issues raised 
in the report 


Strengths 

Overall, college staff have positive attitudes to quality improvement, including 
self-evaluation reporting. Most colleges demonstrate commitment to self-evaluation 
reporting by having strategies for improving it, and comprehensive arrangements for 
implementing it. 

Most college managers and college staff consider that self-evaluation activities and 
reporting have encouraged a culture of reflection and interrogation amongst staff 
which has positively influenced quality improvement. However, most consider that 
evidencing this in self-evaluation reporting is not effective enough. 

The majority of reports address the prompts in quality frameworks within clear local 
contexts. The most specific and detailed self-evaluation reports are produced by 
small teaching or support teams. 


** Jackson, ‘The role of evaluation in self-regulating higher education institutions’, in Jackson (ed) Managing 
Quality and Standards in UK Higher Education (Higher Education Quality Council, 1997) 
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Senior managers usually follow through on actions appropriately allocated to them in 
teaching and support team self-evaluation reports and quality improvement action 
plans, or justify why they are unable to do so. They are most likely to follow through 
on actions which will help the teams realise wider college aims. Effective 
follow-through by senior managers on actions allocated to them leads to college staff 
valuing the process more. 

When college staff are actively involved in continuously improving provision through 
ongoing professional dialogue which underpins and runs parallel to self-evaluation 
reporting, many express the view that professional dialogue has more direct impact 
on improvement in learning and teaching than formal self-evaluation reporting. This 
includes team meetings, staff review and development interviews, internal 
moderation discussions or the conversations which follow peer observation, as well 
as informal discussions amongst staff. They believe that discussions in these 
situations are immediate, often private and focused on solving problems or sharing 
positive experiences quickly. They also consider that the potential adoption of good 
or innovative practices is a highly effective trigger for well-focused professional 
dialogue which analyses current practice. 

More than a few colleges have developed peer review and team teaching systems 
which underpin supportive professional dialogue on learning and teaching. Most 
teaching staff are committed to developing their professional skills through dialogue 
with others, alongside their involvement in structured self-evaluation reporting. 

For most college staff, self-evaluation reporting is valued as a means of 
summarising, evidencing and recording the outcomes of other self-evaluation 
activities, especially professional dialogue. It helps them to reflect on past actions 
and focus on whether they have improved quality, in a structured and systematic 
way. 

Most college staff also feel that self-evaluation reporting provides a useful 
mechanism by which teams can communicate their findings to senior managers. In 
this way teams can provide assurance that they take quality seriously and actively 
seek to improve their services. However, most staff believe that self-evaluation 
reporting of itself is less effective as a means of directly initiating improvement 
because it is usually carried out at such a remove from events. 

College staff are evenly split in their views about the impact of grades on 
self-evaluation reporting. Some staff consider that they enable teams to measure 
the quality of the services they provide to learners and help them aspire to improve. 
However, others feel that discussing and agreeing grades distracts from the main 
purpose of self-evaluation reporting - evaluating the learner experience and planning 
action for improvement. 

Areas for development 

Although most colleges are committed through their strategic planning to 
improvement in self-evaluation reporting, translation into specific actions in 
college-wide staff development plans is unusual in colleges. 
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In spite of colleges’ overall efforts at embedding self-evaluation reporting into quality 
systems, a few have not been able to generate ownership of the process amongst all 
team leaders and staff. A few staff consider it to be solely an exercise in 
compliance, driven either by managers or by HMIE. This is especially so when there 
is little time to carry it out. A few team leaders also consider that an annual cycle is 
too short to carry out meaningful self-evaluation reporting. 

For a majority of college staff an over-prescriptive approach to using quality 
frameworks for self-evaluation reporting constrains their ability to adapt them to their 
own needs and therefore make the outcomes meaningful to their learners. Reports 
which follow the quality framework in an over-prescriptive way can be superficial and 
lack high level evaluations of the learner experience. Most staff, however, agree that 
structure is important and that the SFC/HMIE quality framework provides 
comprehensive and useful prompts for self-evaluation reporting. 


FOCUS ON THE FUTURE 

When colleges have participated In sector-wide Initiatives to Improve learning and 
teaching, for example Focus on Learning 2^^, TESEP (Transforming and Enhancing the 
Student Experience through Pedagogy or the development and Implementation of 
new Scottish Ouallficatlons Authority Higher National qualifications^'^ , this has 
encouraged staff to evaluate their own practices productively, In a forward-looking 
way. Because many staff work In teams to Implement Ideas, they have found being 
observed In class by team members a useful and non-threatening way to evaluate 
the Impact of their new approaches. The discussions which result are often 
described as creative and exciting and staff felt they learned a great deal from them. 


Focus on Learning 2: Steps to Employability ( http://www.sfeu.ac.uk/fol2 
http://www.jisc.ac.uk/whatwedo/programmes/elearning_sfc/sfc_tesep.aspx 
http://www.sqa.org.uk/sqa/41 1 .html 
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6.3 Theme 3: Evidence 

Self-evaluation reporting is effective when college teams gather, analyse and 
respond to key messages from various sources of evidence, including other 
evaluations. 


Effective self-evaluation reporting relies on properly analysed evidence, not simply on the 
opinions of team members or on hearsay. Most importantly, teams must evaluate where 
their strengths and areas for improvement lie based on the interpretation of the key 
messages from gathered evidence. Evidence includes performance indicators (Pis), trend 
patterns and benchmarks, and the views of those who use the college services - learners 
and other stakeholders. Accurate and timely provision of data are essential if it is to be 
useful. Analysis of the underlying reasons for high or low learner performance indicators is a 
particularly important way for teaching teams to identify successes and deficiencies in the 
learner experience. Evidence generated through professional dialogue is particularly useful, 
including where staff assess their own or each others’ performance. Professional dialogue 
and professional reflection provides an important source of evidence about how well staff 
have effected improvement in learning and teaching.^^ 


Strengths 

Almost all colleges have highly effective systems for providing accurate up-to-date 
Pis on learner retention and attainment for teaching teams to analyse in 
self-evaluation reports. 

Most self-evaluation reports by teaching teams at least acknowledge low value Pis. 
Where reports analyse underlying reasons for these in more detail, these sections 
usually demonstrate detailed staff knowledge of individual learners’ circumstances. 

All colleges use a comprehensive range of methods to obtain feedback from 
learners, employers and external bodies. Most teaching teams and support teams 
which impact directly on learners cite learner feedback in the evidence base for their 
self-evaluation reports, and more than a few detail the outcomes of learner 
questionnaires in terms of satisfaction percentage rates. 

A few colleges are piloting recording systems for gathering ongoing evidence from 
professional dialogue and other self-evaluation activities and using these systems to 
provide useful information for further discussions as part of self-evaluation reporting. 

Areas for development 

The majority of self-evaluation reports do not comprehensively analyse underlying 
reasons for low retention and attainment other than learners’ personal circumstances 
or backgrounds, nor do they analyse trends. A few reports do, however, link low 
retention and attainment to ineffective guidance. Detailed analysis of low 
programme or unit attainment is sometimes addressed in self-evaluation reporting on 


Becoming a Critically Reflective Teacher, Stephen D. Brookfield, Jossey-Bass, 1995 
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programme design. Resulting actions include removal of units which are too 
difficult, change in running order of units or integration of units. It is rare for analysis 
of low attainment to consider the impact of the quality of learning and teaching. 

It is unusual for self-evaluation reports to benchmark their Pis against corresponding 
subjects and areas in the rest of the college sector. 

It is very rare for learners to be involved directly in discussion as part of 
self-evaluation reporting, or to have the opportunity to comment on self-evaluation 
reports. 

In most cases, professional dialogue about learning and teaching is not well 
evidenced in self-evaluation reports. It is exceptionally rare for the outcomes of peer 
observation to be included in self-evaluation reports. Most weaknesses relating to 
learning and teaching in self-evaluation reports concern resourcing issues rather 
than learning and teaching issues. 


LINKING EVIDENCE TO EVALUATION 

A few colleges have devised ways of gathering evidence from wider self-evaluative 
activities on an ongoing basis. In one case, staff use quality logs consisting of 
written evidence assembled throughout the academic year. This helps them to keep 
track of formal meetings, informal discussions and learner issues in a methodical 
way. This enables staff not only to cite sources of evidence, but recall what the 
underlying issues were and why they were significant. 


A1 in the SFC/HMIE Quality Framework 

Student Representation in Scottish Further Education Colleges, HMIE, 2006 

The final report from the SFC Joint Quality Review Group gives learner engagement a very high profile as 
one of its three key principles of an effective quality system, SFC, 2007, (see foreword) 


21 



Effective self-evaluation reporting in Scotland’s colleges 


6.4 Theme 4: Action plans 

Self-evaluation reporting is effective when it generates a clear agenda for team 
action (a quality improvement action plan) aimed at improving services to 
learners. 


The generation of a dear, agreed agenda for action with specific, measurabie, achievabie, 
reaiistic and time-bound (SMART) targets is the most important outcome of seif-evaiuation 
reporting. Quaiity improvement action pians are oniy fuiiy effective when they target aii 
major areas for improvement and actions are aiiocated to specific peopie. Of course, action 
pians must not just be agreed but acted upon and actuaiiy resuit in improvements to 
services. The reguiar monitoring of progress on specified actions, whether within the team 
itseif or by senior coiiege managers, is criticaiiy important to them being carried out 
effectiveiy. It is important that coiieges evaiuate the impact of previous actions, and do not 
just record that they have taken piace. Oniy then can coiieges be certain that seif-evaiuation 
reporting has been effective and has resuited in improvement. 


Strengths 

Almost all self-evaluation reports result in an action plan with objectives for 
improvement. Most targets are assigned to specific members of staff for action, 
though not always members of the reporting team. Most have timescales for 
achievement.. 

Action plans are mostly well monitored in team meetings or more widely in meetings 
with senior management. Most actions are carried out to the time specified, or 
teams record justifications for their non-completion. 

The outcomes of previous action plans usually feature in self-evaluation reports. 
However, in most cases, they are simply recorded as being ‘achieved’, ‘ongoing’ or 
‘carried over’. 

Areas for development 

Although specified actions in quality improvement plans are usually attributed to 
identified members of staff, timescales can sometimes be vague, often described as 
‘ongoing’, and most have objectives which are not expressed as specific, 
measurable targets. In more than a few plans, objectives are very vague and use 
phrases such as ‘continue to consider strategies for improvement in retention’, or 
‘investigate ways of implementing action learning where possible’. 

The majority of quality improvement action plans do not specifically address low 
levels of retention and attainment evidenced by low Pis. They rarely set out specific 
strategies to improve learning and teaching. 

The impact of actions taken to bring about improvements to the learner experience is 
rarely evaluated specifically. 
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6.5 Theme 5: College-wide approaches 

Self-evaluation reporting is effective when college systems support it and 
colleges use it to gather intelligence for whole-college planning. 


Well-coordinated college systems for managing self-evaluation reporting are essential to its 
success. These include supportive monitoring by senior management and coordination of 
team and college planning systems. Well-thought-out proformas, clear guidelines, the 
provision of time for self-evaluation activities and reporting and well-targeted training and 
support are all helpful. Self-evaluation reporting helps to address wider college strategic and 
operational aims when resulting local action plans dovetail with college ones. Such 
dovetailing provides colleges with assurance that college aims have been understood by 
operational teams. Analysing accurate team self-evaluation reports allows senior managers 
to systematically identify the health of the constituent parts of the organisation through the 
filter of operational expertise. They can identify where good practice, strengths and areas 
for improvements lie, where resources and support should be targeted, or where changes in 
services need to be made. They can also monitor how well self-evaluation reporting is 
carried out and see where skills in it need to be better developed amongst team leaders and 
staff. Self-evaluation reports which evaluate services in the light of developments in the 
external operating environment provide senior managers with useful business planning 
intelligence. When college managers carry out college self-evaluation reporting, key 
messages from teaching and support teams provide an important evidence base. 


College planning cycle 


COLLEGE STRATEGIC AND 
OPERATIONAL PLAN 




• MONITORING 

• OUALITY ASSURANCE 

• GATHERING INTELLIGENCE 


DEPARTMENT OPERATIONAL PLANS 


SELF-EVALUATION ACTIVITIES 

(PAGE 2) 


SELF-EVALUATION 

REPORTING 
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Strengths 

HMIE college review reports awarded grades of very good for quality assurance in 
45% of cases and good in 48% during the current review cycle (January 2005 to 
April 2007). The relevant sections of these reports highlight not only the 
effectiveness of self-evaluation reporting systems, but also the effectiveness of other 
self-evaluation activities and quality systems. 

Self-evaluation reporting cycles are usually well timed to gather evidence from other 
self-evaluation activities and quality processes including programme team meetings, 
learner feedback events, internal and external moderation, and internal audit. 

Almost all colleges are actively engaged in improving self-evaluation activities and 
reporting, through the involvement of external support agencies and through internal 
support systems. A few colleges have successfully piloted new SQA professional 
development units^® which aim to improve staff performance in self-evaluation 
reporting. 

Colleges are almost always highly supportive of their staff and focused on 
developing them to their fullest potential, even if staff are experiencing difficulties. 
Most colleges strive to generate non-blaming quality cultures. When they get 
feedback on self-evaluation reports from senior or quality managers, staff mostly find 
it clear, constructive and influential on how well they carry out self-evaluation 
reporting in the future. 

More than a few college-produced guidelines are comprehensive and thoughtful. 
Proformas for self-evaluation reporting are usually clear and easy to use. 

Most team quality improvement action plans address college-wide operational 
planning aims, albeit indirectly. Departmental operational plans are often 
aggregated from team action plans and usually address college operational aims 
more directly. College proformas for departmental operational plans are frequently 
designed to link them to college operational plans. 

Where monitoring meetings take place, senior college managers usually use them 
effectively to support college teams and to discuss issues about internal and external 
operating environments constructively with staff. Most senior managers find these 
meetings helpful for identifying improvement strategies with staff and for targeting 
support and resources. Most senior managers also find them useful for identifying 
and disseminating good practice amongst college teams. 

A few colleges carry out college management self-evaluation reporting 
systematically as part of their own planning, monitoring and improvement process. 
Where this happens, evidence is usually gathered effectively from a wide range of 
sources, including appropriate team leaders, staff and external stakeholders. These 
reports usually demonstrate a well-coordinated approach to identifying and 
addressing college-wide weaknesses. 


PDA Advanced Diploma in Quality Improvement, SQA, 2007 
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Most senior managers are well aware of where these are deficiencies in the quality 
of self-evaluation reporting within colleges. Most quality managers act effectively on 
this knowledge to provide additional support. 

Areas for development 

In a few cases training and support is not sufficiently targeted at those who find 
self-evaluation reporting difficult or who are new to it. 

The quality of self-evaluation reports directly influences how well they can provide 
intelligence for senior managers. Those which are not comprehensive, sufficiently 
evaluative or based on evidence contribute little to wider college planning. 

Most college management self-evaluation reports (against the ‘B’ elements) are 
written as a precursor to HMIE review and not as part of a regular and systematic 
process. It is rare for these reports to cite major concerns. 

College management self-evaluation reports do not always include input from a wide 
enough range of sources for resulting outcomes to enjoy widespread staff 
ownership. It is also rare for the resulting reports to be systematically and 
comprehensively shared with staff. 

Although senior managers feel they have widespread knowledge of the internal and 
external operating environment through professional dialogue with colleagues, most 
colleges do not carry out formal analysis of common themes in self-evaluation 
reports to inform wider college strategic planning. 


COLLEGE GUIDELINES 

Several colleges have drawn up useful guidelines for their staff to help them carry out 
self-evaluation reporting. A few colleges use self-evaluation reporting proformas with quaiity 
indicators reconfigured as questions. Staff find this heips them to take a more holistic 
approach to the prompts in the quaiity framework and see them as starting points for 
discussion rather than a checklist for compiiance. 


SELF-EVALUATION AND PLANNING 

In one college, staff were able to input information to an oniine database which was 
configured so that it directiy linked departmentai seif-evaiuation reporting outcomes and 
action pians to the coiiege strategic aims. The oniine faciiity aiiowed staff at aii sites to 
access the database and view reports, pians and progress for aii coiiege areas. Managers 
couid aiso use this up-to-date information to produce coiiege reports and inform team 
meetings. This joined-up approach to self-evaluation reporting heiped staff contextuaiise it 
within wider coiiege aims. 
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6.6 Theme 6: Reporting 

Effective seif-evaiuation reporting enabies coiiege teams to generate reports 
which provide assurance on whether they are effective at continuousiy 
improving their services. 


Self-evaluation reporting has to be carried out properly for it to be effective and valued, and 
for its findings to be relied on as a basis for quality improvement. It therefore has to be 
thorough and self-critical. It has to cover all the relevant ground, genuinely reflect the 
experience of learners and be based on sound evidence. Most importantly, in order for 
self-evaluation reporting to have meaning, it has to actually evaluate services and not simply 
describe them. Without this filter of interpretation, it cannot provide assurance that college 
teams fully understand what the evidence tells them nor that they know whether they are 
responding to its messages. If this happens, self-evaluation reporting becomes a marginal, 
compliance-driven process. Self-evaluation reports have to be written in a way which 
demonstrates that the self-evaluation reporting process has been carried out effectively. 
Reports need to communicate their findings usefully to different people. They have to be 
detailed, clear and meaningful, otherwise they are difficult to moderate, whether by internal 
or external bodies. Reports also have to be recognisable to those team members who 
contributed to the self-evaluation, and reflect the shared discussions. The skills of those 
writing these important reports are critical to their impact. 


Strengths 

HMIE college review reports have awarded grades of very good \n 34% of cases and 
good in 51% for quality improvement during the latest review cycle. They almost 
always highlight staff commitment to improvement and how colleges have effected 
improvement overall. 

There are more than a few examples of very well-constructed and very 
well-evidenced self-evaluation reports. They are detailed and comprehensive, and 
the product of shared input. These reports honestly identify appropriate strengths 
and areas for improvement and result in agreed action plans with SMART targets. 
Such examples are often found in colleges which carry out a range of highly effective 
self-evaluation activities and have comprehensive monitoring systems and 
well-targeted support for staff. In most of these colleges, services to learners have 
improved over time, though it is rarely evidenced how much of the improvement is 
directly attributable to self-evaluation reporting and quality improvement action 
planning. 

Areas for development 

Although there are examples of well written self-evaluation reports, the majority 
neither give enough detail nor have sufficient clarity to provide full assurance about 
the quality of services or that quality improvement is being well enough addressed. 

A few reports do not adequately justify their grade profiles. It is unusual for teaching 


www.hmie.gov.uk 
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teams, or college management teams particularly, to be self-critical in self-evaluation 
reports and to award themselves grades of fair or unsatisfactory, even where Pis 
and other evidence indicate that there are serious problems. Where college teams 
record grades of fair or unsatisfactory, it is usually in the area of resources. 

The majority of reports are written descriptively, not evaluatively. In overly 
descriptive reports, the links between text, identified areas for development and 
action plans are often tenuous. 

Quality improvement action plans are only as effective as the underlying 
self-evaluation activities and reports. In a majority of cases, insufficient evaluation or 
analysis of underpinning evidence in the self-evaluation report results in action plans 
that do not effectively address weaknesses, particularly in retention and attainment. 
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7. Recommendations 

Colleges should: 

■ encourage and facilitate self-evaluation activities such as professional 
dialogue, especially on learning and teaching, retention and attainment; 

■ ensure that self-evaluation reports and quality improvement action plans 
record the impacts that previous actions arising from these processes have 
had on improving quality for learners and other stakeholders; 

■ ensure that self-evaluation reporting by teaching teams focuses on learning 
and teaching, and retention and attainment, and involves the rigorous analysis 
of evidence, including Pis; 

■ ensure that self-evaluation reporting is evaluative rather than descriptive; 

■ ensure that quality improvement action plans use SMART targets; and 

■ provide targeted training and support to improve evaluative skills and 
self-evaluation report writing, using appropriate training materials such as the 
PDA Advanced Diploma in Quality Improvement. 

SFC, in consultation with the college sector and HMIE, should: 

■ explore the benefits of developing a review model which moderates how 
effectively colleges are using self-evaluation activities and reporting to provide 
assurance of ongoing quality improvement in colleges. 


HMIE ACTIONS 
HMIE will: 

■ develop with SFC and in consultation with the college sector, a revised quality 
framework for Scotland’s colleges and associated guidance which helps 
college teams to contextualise the self-evaluation process and generate 
innovative and effective approaches to it; 

■ encourage self-evaluation activities such as professional dialogue, especially 
on learning and teaching, retention and attainment as part of existing 
follow-through arrangements and future engagement with colleges; and 

■ disseminate exemplars of good practice in self-evaluation reporting. 


28 


Effective self-evaluation reporting in Scotland’s colleges 


Appendix 1 


Other models of self-evaluation 

England 

Self-evaluation - or self-assessment as it is more commonly known - sits very much 
at the heart of inspection models in England. The requirement in England in relation 
to self-assessment is much more explicit than in Scotland. A recent report published 
by Ofsted in July 2006 - Best practice in seif-evaiuation, A survey of schoois, 
coiieges and iocai authorities^^ - highlighted that: 

‘Colleges have a relatively long history of self-evaluation since they became 
independent corporations in 1993. They have been expected to develop 
quality assurance systems based on a self-critical assessment of their 
provision. In 2001 , when Ofsted and the Adult Learning Inspectorate became 
responsible for inspecting colleges, a college’s self-assessment report, 
together with data on students’ outcomes, became the starting point for 
inspection. Since 2005, college inspection has been matched more closely to 
the quality of provision, with good colleges being inspected with a lighter 
touch. Such arrangements therefore place more emphasis on a college’s 
self-assessment.’ 

Providers are required to complete an annual self-assessment report that evaluates 
all aspects of their provision, accredited and non-accredited. This generally comes 
from their continuous process of self-assessment. The chief purpose of 
self-assessment is to support the provider’s own work on quality improvement and to 
measure progress against its own mission and goals. 

The recently revised approach of the planning and funding bodies, and the 
inspectorates now places much greater emphasis on risk assessment, based on an 
evaluation of the capacity of the provider to maintain and improve quality. 

A climate of trust is required in which a provider’s staff are constructively self-critical 
about their performance. Providers are required to demonstrate understanding of 
what they do well, what needs improving and how improvement can be monitored, 
achieved and evaluated. According to the Learning and Skills Council,^® the key test 
of the resulting self-assessment report is its ability to demonstrate how high quality is 
sustained and improvement is ensured. 


www.ofsted.ac.uk 

www.lsc.gov.uk 
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Quality Assurance Agency for Higher Education (QAA) Scotland 

From 2003 to 2007, QAA Scotland operated an Enhancement-Led Institutional 
Review model (ELIR).^° This followed extensive consultation which concluded that 
in general: 

‘Scottish higher education institutions had in place effective quality 
management systems relating to the experience of students and the 
standards of their awards, and that the subject provision experienced by 
students was highly satisfactory or better.’ 

The model was operated as part of the overarching Quality Enhancement 
Framework containing five elements: subject review coordinated by the HEIs; 
student engagement in quality management; an agreed set of public information; a 
national programme of enhancement themes and EUR. It was also operated through 
a partnership between QAA Scotland, the Scottish Higher Education Funding 
Council (formerly SHEFC, now SFC), Universities Scotland, the National Union of 
Students (NUS) and the Coalition of Higher Education Students in Scotland 
(CHESS). 

This has resulted in a change in the focus of external review to go beyond how 
institutions quality assure provision for students to encompass also how they actively 
enhance it. An important part of the review model up to 2007 was a cycle of subject 
reviews carried out internally, following SHEFC guidelines (which stated that the 
cycle should not be longer than six years). An institution’s operation of subject 
reviews was considered during EUR, which itself took place every four years and 
was supported by annual discussions between QAA officers and institutional 
representatives. 

The starting point for each EUR was the institutional reflective analysis, which 
evaluated the institutions’ ability to learn from what had happened in the past in order 
to inform the future. Institutions were expected to address three key questions: 
‘where are we now?’, ‘where do we want to be in the future?’ and ‘how are we going 
to get there?’ They were also expected to identify the range of reference points they 
used within their own quality systems and procedures, including the QAA’s own code 
of practice, the views of external professional bodies and, very importantly, the views 
of students. Case studies were appended to the reflective analysis which 
demonstrated the institution’s strategic approach to quality enhancement. The 
reflective analysis provided EUR teams with essential information which helped 
them to scope further enquiry during the review visits. 

At subject level, institutions were required to provide assurance that they annually 
monitored the academic standards of awards and quality of provision. They were 
expected to devise systems in which the outcomes of subject level self-evaluation 
were considered, within departments and schools, and used to inform institutional 
strategic planning. Each institution devised its own methodology for doing this, 
which was expected to adhere to the relevant section of the QAA Code of Practice. 


www.qaa.ac.uk/reviews/ELIR/handbook/Scottish_hbook_preface.asp 
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This subject area report was drawn from a range of sources, including performance 
indicators and feedback from students. 

In the colleges which are partners in the UHI Millennium Institute (UHI), all provision 
from SCQF Level 7 and above was reviewed under the QAA model between 2000 
and 2006. Provision below that level was reviewed by HMIE. UHI also participated 
in an EUR in 2007.^^ 


The European Foundation for Quality Management (EFQM) Excellence Model 

The EFQM Excellence Model was published in 1991 as an umbrella framework 
against which organisations could assess their services and identify ways of 
improving them. The Excellence Model is primarily a framework for self-assessment 
against nine criteria, divided into two groups, enablers and results, which taken 
together, are designed to provide organisations with key areas to review and 
improve. There are different techniques for carrying out self-assessment which 
help provide different levels of detail. The EFQM Excellence Model is used by more 
than 20,000 European private and public sector organisations, including two of 
Scotland’s colleges. Qrganisations can apply for awards from EFQM at different 
levels in recognition of how far they have developed against specific quality criteria. 

The Excellence Model is primarily aimed at making companies more competitive, but 
its basic principles have been adapted for educational and other public sector 
self-evaluation in Scotland. HMIE has developed an overarching quality framework 
based on the EFQM model which provides a systematic structure for self-evaluation 
over nine key themes, and addresses a tenth theme, the evaluation of the overall 
capacity for improvement, at the end of this process. This framework has been 
adapted for use in other education sectors, including schools, community learning 
and development and the education functions of local authorities.^^ Qther public 
sectors, including social work and the police are also in the process of developing 
quality frameworks based on the EFQM. 


' www.uhi.ac.uk 

European Framework for Quality Management, www.qualityscotland.co.uk 
How good is our school?: the Journey to Excellence, HMIE, 2007 
How good is our community learning and development?2, HMIE, 2006 
Quality Management in Education, HMIE, 2006 
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Appendix 2 


Colleges involved in discussions for this report 

• Lauder College 

• Angus College 

• Clydebank College 

• West Lothian College 

• Dumfries and Galloway College 

• Perth College 

• Lews Castle College 

• Glasgow College of Nautical Studies 

• Elmwood College 

• Cardonald College 

Other organisations consuited: 

• Scottish Further and Higher Education Funding Council 

• Scottish Further Education Unit 

• Association of Scottish Colleges 

• UHI Millennium Institute 

• Ofsted/Adult Learning Inspectorate 

• Quality Assurance Agency for Higher Education in Scotland 
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Appendix 3 

Standards and quality in Scottish Further Education (SFC/HMIE) 

Quality framework for Scottish FE colleges 
May 2004 

Area A: Curriculum resources, processes and outcomes 


Elements: 

A1 

Programme design 

A2 

Accommodation for learning and teaching 

A3 

Equipment and materials 

A4 

Staff 

A5 

Learning and teaching process 

A6 

Assessment 

A7 

Learner progress and outcomes 

A8 

Guidance and learner support 

A9 

Quality assurance and improvement 


Area B: Leadership and quality management 


Elements: 

B1 

Educational leadership, direction and management 

B2 

Access and inclusion 

B3 

Guidance and support 

B4 

Resources and services to support the learner 

B5 

Staff 

B6 

Quality assurance 

B7 

Quality improvement 
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Appendix 4 


Analysis of grades 


It is not formally part of the HMIE review model to validate college’s self-evaluation 
grades. However, evidence from a sample of 24 HMIE college reviews carried out 
between January 2005 and June 2007 was analysed to identify patterns of 
concurrence between self-evaluation reporting grades and HMIE grades. 

The sample focused on college management team and teaching team 
self-evaluation reports ie the B elements (Leadership and Quality Management), 
A5 (Learning and Teaching Process) and A7 (Learner Progress and Outcomes). 


B Elements: Leadership and quality management 

% of HMIE grades lower than self-evaluation grades 27 

% of HMIE grades higher than self-evaluation grades 8 

% of concurrence between HMIE and self-evaluation grades 65 


A5 Learning and teaching process 

% of HMIE grades lower than self-evaluation grades 1 6 

% of HMIE grades higher than self-evaluation grades 1 0 

% of concurrence between HMIE and self-evaluation grades 74 


A7 Learner progress and outcomes 

% of HMIE grades lower than self-evaluation grades 20 

% of HMIE grades higher than self-evaluation grades 6 

% of concurrence between HMIE and self-evaluation grades 74 
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